Moving picture coding, coded-moving picture bitstream conversion and coded-moving picture bitstream multiplexing

ABSTRACT

First pictures as reference for inter-picture prediction of an incoming moving-picture and second pictures are coded. The first pictures are coded by intra-picture or unidirectional inter-picture predictive coding. The second pictures are coded by bidirectional inter-picture predictive coding by referring the first or their locally-decoded pictures to obtain a moving-picture bitstream. The bitstream is multiplexed with obtained motion activity information. Incoming first bitstreams are multiplexed into a bitstream at a rate lower than a total bit rate of the incoming first bitstreams. Each bitstream has first bitstreams of the first and second pictures. A code amount of the multiplexed bitstream is obtained for each reproduction period as virtual buffer occupancy, according to which a decimation rate is set on the second pictures. The second pictures bitstreams are decimated from the incoming first bitstreams according to the decimation rate to obtain second bitstreams, which are multiplexed to obtain the multiplexed bitstream.

This is a divisional application of Ser. No. 09/812,784 filed Mar. 15,2001 now U.S. Pat. No. 6,954,499.

BACKGROUND OF THE INVENTION

The present invention relates to highly efficient coding of movingpictures into a bitstream at a small code amount for efficient videodata transfer, storage and displaying, such as, MPEG-coding withinter-picture predictive coding, conversion of coded-moving picturebitstream at a different transfer rate, and also multiplexing of severalcoded-moving picture bitstreams into one bitstream.

Pictures under MPEG-coding are divided into three different types I-, P-and B-pictures. I-pictures (intra-coded pictures) are codedindependently, entirely without reference to other pictures. P-pictures(unidirectionally predictive-coded pictures) are compressed by codingthe differences the pictures and reference preceding I- or P-pictures.B-pictures (bidirectionally predictive-coded pictures) are alsocompressed by coding the differences the pictures and referencepreceding or upcoming I- or P-pictures.

A coded B-picture bitstream can be removed from a coded bitstreambecause B-pictures are not used as reference pictures.

A coded bitstream of 30 frames/second with P-pictures for every 3frames, for example, can be converted into a bitstream of 10frames/second by removing B-picture streams only. The 10-frame/secbitstream has a low transfer bit (code) rate in accordance with theremoved B-picture amount.

In detail, a moving picture bitstream is supplied to a decimator toremove B-picture bitstreams according to decimation control data. TheB-picture-removed bitstream is supplied to a buffer intermittently foreach period in which B-pictures have been removed. The buffer holds suchan intermittent input and outputs a sequential bitstream at a lowtransfer bit rate in accordance with the removed B-picture amount.

Moving pictures reproduced from such an output bitstream however cannotmove smoothly on screen due to decrease in frame rate of the reproducedpictures which is caused by B-picture removal without respect tomovement of original moving pictures.

In digital broadcasting, moving picture bitstreams on several channelsare multiplexed into one bitstream suitable for a transfer line. Atransfer bit rate of the multiplexed bitstream is basically the total ofthe moving picture bitstreams of the several channels.

An amount of codes required for obtaining a desired reproduced picturequality varies according to portions (periods) of moving pictures inmoving picture coding. A transfer bit rate is set at the maximum codeamount if it is a fixed rate. Portions (periods) of moving pictures thatrequire a high transfer bit rate vary according to channels formultiplexing in several channels. A fixed transfer bit rate at themaximum code amount after multiplexing thus becomes smaller than thetotal of the maximum code amounts at the channels.

The more the channels to be multiplexed, the smaller the fixed transferbit rate after multiplexing. This tendency lowers a transfer bit rate ontransfer line by multiplexing. Each channel, however, carries abitstream at a variable transfer rate, thus requiring a rate control,which results in an overall rate control over the several channels.

Such an overall rate control is achieved by a quantization control forevery channel based on buffer occupancy on a virtual buffer providedafter multiplexing. A reference value, that is, a quantization scalefactor on quantization step width based on the buffer occupancy isreturned to an encoder for each channel.

In detail, moving picture inputs on several channels are coded byMPEG-encoders, respectively, to bitstreams. The bitstreams on theseveral channels are multiplexed by a multiplexer into one MPEG-2transport stream (ST).

The multiplexed bitstream is supplied to a code amount observer having avirtual buffer that corresponds to a buffer of an MPEG-decoder. The codeamount observer observes buffer occupancy on the virtual buffer bydetecting a frame code amount of a bitstream stored in the virtualbuffer at decoding timing. The more the each frame code amount, the lessthe buffer occupancy, in other words, the less the each frame codeamount, the more the buffer occupancy.

Information on buffer occupancy (BO) is supplied by the code amountobserver to a rate (quantization) controller. A reference value, thatis, a quantization scale factor on quantization step width is output bythe rate controller based on the BO information and returned to theMPEG-encoders for the several channel. The quantization scale factor isused for all MPEG-encoders. The less the buffer occupancy, the largerthe quantization scale factor, in other words, the more the bufferoccupancy, the smaller the quantization scale factor.

Each MPEG-encoder is controlled by the quantization scale factor onqaunization step width. The finer the quantization, the larger the codeamount generated by each MPEG-encoder, in other words, the more coarsethe quantization, the smaller the code amount generated by theMPEG-encoder.

The feed-back control on quantization scale is performed to the totalcode amount for all channels to achieve stable quantization in whichcode amounts are large for some channels but small for the otherchannels to a constant total code amount.

When several MPEG-encoders are controlled in the same way, they generatea large amount of codes for pictures that, for example, move rapidly onscreen, thus requiring a high transfer rate whereas a small amount ofcodes for pictures that, for example, move very slowly, thus requiring alow transfer rate. Codes are thus shifted from channels that require alow amount of codes to other that require a high amount of codes.

Such multiplexing as described above requires a variable transfer ratealso on a transfer line connecting MPEG-encoders and a multiplexer withfeed-back control of a quantization scale factor to the encoders.

This type of processing is performed with a unit containing bothMPEG-encoders and a multiplexer for high-speed transfer therebetween. Itis difficult to multiplex bitstreams that have been coded at andtransferred from a remote station. Therefore, in a system having acentral station and a remote station, video signals with no compressionare transferred from the remote station to the central station in whichthe video signals are encoded and multiplexed.

As discussed, an overall control of generated codes is required at theMPEG-encoder side, which is thus difficult when encoders are located inseveral remote places. Moreover, such an overall code amount control isimpossible when multiplexing bitstreams that have already been encodedat remote places.

SUMMARY OF THE INVENTION

A purpose of the present invention is to provide an apparatus and amethod of encoding moving pictures and converting a moving picturebitstream into another at a transfer rate by which moving pictures movesmoothly on screen when decoded.

Another purpose of the present invention is to provide an apparatus anda method of multiplexing bitstreams of moving pictures that have alreadyencoded with an optimum code amount reduction.

The present invention provides a coding apparatus for coding firstpictures, that are set at a predetermined interval, to be used asreference pictures for inter-picture prediction of an incoming movingpicture and coding second pictures different from the first pictures,the apparatus including: an encoder to encode the first pictures byintra-picture coding or unidirectional inter-picture predictive coding,and to encode the second pictures by bidirectional inter-picturepredictive coding using the first pictures or locally-decoded picturesof the first pictures as the reference pictures, thus obtaining amoving-picture bitstream; a motion amount detector to obtain motionactivity on the incoming moving picture; and a multiplexer to multiplexthe moving-picture bitstream and the information of motion activity.

Moreover, the present invention provides a bitstream conversionapparatus for converting an incoming first moving-picture bitstream at afirst code transfer rate to a second moving-picture bitstream at asecond code transfer rate, the incoming first moving-picture bitstreamhaving been obtained by coding first pictures set at a predeterminedinterval and used as reference pictures for inter-picture prediction ofa moving picture to be coded and coding second pictures different fromthe first pictures, the apparatus including: a motion amount detector toobtain motion activity on the incoming first moving-picture bitstream; adecimation controller to set a decimation rate on the second pictures ofthe incoming first moving-picture bitstream according to at least themotion activity; and a bitstream decimator to decimate only bitstreamsof the second pictures from the incoming first moving-picture bitstreamat another predetermined interval according to the decimation rate.

Furthermore, the present invention provides a bitstream multiplexingapparatus for multiplexing a plurality of incoming first moving-picturebitstreams into a moving-picture bitstream at a bit rate lower than atotal bit rate of the incoming first moving-picture bitstreams, eachincoming first moving-picture bitstream having first moving-picturebitstreams of first pictures used as reference pictures forinter-picture prediction and bitstreams of second pictures other thanthe first pictures, the apparatus including: a decimation controller toset a decimation rate on the second the pictures of the incoming firstmoving-picture bitstreams according to virtual buffer occupancy to themultiplexed moving-picture bitstream; a bitstream decimator to decimatethe bitstreams of the second pictures from the incoming firstmoving-picture bitstreams at a predetermined interval according to thedecimation rate, thus obtaining second bitstreams; a multiplexer tomultiplex the second bitstreams to obtain the multiplexed bitstream; andvirtual buffer means for obtaining a code amount of the multiplexedbitstream for each reproduction period as the virtual buffer occupancy.

Moreover, the present invention provides a method of coding firstpictures, that are set at a predetermined interval, to be used asreference pictures for inter-picture prediction of an incoming movingpicture and coding second pictures different from the first pictures.The first pictures are coded by intra-picture coding or unidirectionalinter-picture predictive coding, and the second pictures are coded bybidirectional inter-picture predictive coding using the first picturesor locally-decoded pictures of the first pictures as the referencepictures, thus obtaining a moving-picture bitstream. Motion activity isobtained on the incoming moving picture. The moving-picture bitstreamand the information of motion activity are multiplexed.

Moreover, the present invention provides a method of converting anincoming first moving-picture bitstream at a first code transfer rate toa second moving-picture bitstream at a second code transfer rate, theincoming first moving-picture bitstream having been obtained by codingfirst pictures set at a predetermined interval and used as referencepictures for inter-picture prediction of a moving picture to be codedand coding second pictures different from the first pictures. Motionactivity is obtained on the incoming first moving-picture bitstream. Adecimation rate is set on the second pictures of the incoming firstmoving-picture bitstream according to at least the motion activity.Bitstreams of the second pictures only are decimated from the incomingfirst moving-picture bitstream at another predetermined intervalaccording to the decimation rate.

Furthermore, the present invention provides a method of multiplexing aplurality of incoming first moving-picture bitstreams into amoving-picture bitstream at a bit rate lower than a total bit rate ofthe incoming first moving-picture bitstreams, each incoming firstmoving-picture bitstream having first moving-picture bitstreams of firstpictures used as reference pictures for inter-picture prediction andbitstreams of second pictures other than the first pictures. A codeamount of the multiplexed bitstream is obtained for each reproductionperiod as virtual buffer occupancy. A decimation rate is set on thesecond pictures of the incoming first moving-picture bitstreamsaccording to the virtual buffer occupancy. The bitstreams of the secondpictures are decimated from the incoming first moving-picture bitstreamsat a predetermined interval according to the decimation rate, thusobtaining second bitstreams. The second bitstreams are multiplexed toobtain the multiplexed bitstream.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows a block diagram of a motion-compensated coding apparatus asa preferred embodiment of a moving picture coding apparatus according tothe present invention;

FIG. 2 shows a block diagram of the first preferred embodiment of amoving picture bitstream conversion apparatus according to the presentinvention;

FIG. 3 illustrates decimation of B-picture according to a decimationrate (frame rate) decided by the moving picture bitstream conversionapparatus shown in FIG. 2;

FIG. 4 illustrates decimation rate (frame rate) control performed by themoving picture bitstream conversion apparatus shown in FIG. 2;

FIG. 5 illustrates frame rate control with two value of buffer occupancyperformed by the moving picture bitstream conversion apparatus shown inFIG. 2;

FIG. 6 shows a block diagram of the second preferred embodiment of amoving picture bitstream conversion apparatus according to the presentinvention;

FIG. 7 shows a block diagram of the first preferred embodiment of amoving-picture bitstream multiplexing apparatus according to the presentinvention;

FIG. 8 shows a block diagram of the second preferred embodiment of amoving-picture bitstream multiplexing apparatus according to the presentinvention;

FIG. 9 illustrates decimation control according to motion activity andbuffer occupancy obtained in the second embodiment shown in FIG. 8; and

FIG. 10 illustrates B-picture decimation according to motion activityobtained in the second embodiment shown in FIG. 8.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Preferred embodiments according to the present invention will bedisclosed with reference to the attached drawings.

A concept of coding according to the present invention is that adecimation rate (or picture rate) on B-pictures is controlled inaccordance with motion activity of moving pictures.

In one embodiment, bitstreams of B-pictures that are not used asreference frames for predictive coding are less decimated for movingpicture portions of high motion activity whereas decimated more formoving picture portions of low motion activity.

This processing keeps high picture quality which would otherwise belowered due to decimation of B-pictures. This is based on the followingfacts: According to human visual characteristics, it is a problem formoving picture portions of high motion activity that moving picturesmove not smoothly on screen when decoded due to decrease in picturerate. This is, however, not a problem for moving picture portions of lowmotion activity.

(Moving Picture Coding)

Disclosed first with reference to FIG. 1 is a motion-compensated codingapparatus as a preferred embodiment of a moving picture coding apparatusaccording to the present invention.

A moving-picture video signal is supplied to a switch 2 and a motionvector (MV) detector 7 through an input terminal 1. The input videosignal carries progressive moving pictures of 60 fps (frame per second),or 480 p (p: progressive). The switch 2 and another switch 16 are turnedon/off concurrently for supplying one I- or P-picture to a subtracter 3for every sixth frame and the remaining B-pictures to a picture memory8.

B-pictures have been stored in the picture memory 8 until coding-of thepreceding I- or P-picture is finished. In this embodiment, 5-frameB-pictures have been stored in the picture memory 8 for a 6-frame perioduntil coding of one I- or P-picture per sixth frame is finished, andsupplied to the subtractor 3 via the switch 16.

A predictive signal supplied by a motion-compensated interframepredictor 10 is subtracted from each I- or P-picture video signal by thesubtracter 3 for generation of a predictive error signal. The predictiveerror signal is supplied to a discrete cosine transformer (DCT) 4.I-picture video signals are supplied to the DCT4 as they are because ofno predictive error signal. The predictive error signal is transformedinto DCT coefficients by DCT-processing and supplied to a quantizer 5.The DCT coefficients are quantized at a predetermined quantization stepwidth to become fixed-length codes and supplied to a variable-lengthencoder 6. The fixed-length DCT coefficients (predictive error signal)are compressed by using variable-length codes and supplied to amultiplexer 13. The coded predictive error signal is multiplexed with amotion activity (MA) signal supplied by a motion amount detector 15 tobecome a bitstream. The bitstream is output through an output terminal14.

The fixed-length DCT coefficients for I- or P-pictures are furthersupplied by the quantizer 5 to an De-quantizer 9 and then an inverse-DCT12. The De-quantizer 9 and the inverse-DCT 12 perform processing that isinverse of processing performed by the DCT 4 and quantizer 5, toreproduce the predictive error signal.

The reproduced predictive error signal is supplied to an adder 11 andadded to a predictive signal, to reproduce a moving picture signal(locally decoded moving picture). The reproduced moving picture signalis supplied to the motion-compensated interframe predictor 10.

The interframe predictor 10 stores the reproduced moving picture signalfor I- or P-pictures as a reference picture signal and performs motioncompensation in accordance with a motion vector (MV) detected by the MVdetector 7, to generate a predictive signal which is then supplied tothe subtracter 3 and adder 11.

Predictive signal generation depends on I-, P- and B-pictures. Indetail, no predictive signal is generated for an incoming I-picture; apredictive signal is generated for an incoming P-picture based on apreceding I- or P-picture (unidirectional prediction); and a predictivesignal is generated for an incoming B-picture based on preceding and/orupcoming I- or P-pictures (bidirectional prediction).

Motion activity detection performed by the MV detector 7 and motionamount detector 15 is disclosed in detail.

The MV detector 7 detects a spatial movement amount of a reference framewith respect to a frame to be coded for every (16×16) blocks. Thedetected spatial movement amount is supplied as a motion vector (MV) tothe motion-compensated interframe predictor 10 and motion amountdetector 15. The motion vector is detected for both I- and P-pictures inthis embodiment, although no motion vector is detected for I-pictures inwell-known coding apparatus.

The motion amount detector 15 detects motion activity based on theincoming motion vector. The detected motion activity is supplied to themultiplexer 13. Motion activity detection is performed per 0.1 secondfor each segment between a P (I)-picture and the next P (I)-picture.

The motion vector used for motion activity detection is a motion vectorused for P-picture prediction. In detail, motion activity is detected byusing a horizontal component MVx (i, j) and a vertical component MVy (i,j) of a motion vector used for P-picture coding in which “i” and “j”indicate horizontal and vertical positions, respectively, on one framewhen movement of one pixel is 1.0.

Motion activity MA1 for the entire frame is obtained according to thefollowing equation (1).

$\begin{matrix}{{MA1} = {\left( {{\sum\limits_{i = 0}^{44}{\sum\limits_{i = 0}^{29}{{MVx}\left( {i,j} \right)}^{2}}} + {{MVy}\left( {i,j} \right)}^{2}} \right)/1350}} & (1)\end{matrix}$

Another technique to detect motion activity is deciding how many blockson which pictures portions move rapidly on screen. This is based on thefact that a picture portion that is moving rapidly on screen is verynoticeable even though an entire picture is not moving.

In detail, motion activity MA2 for the entire frame is obtained based ona degree of motion on each block MVD(i, j) according to the followingequations (2) and (3).MVD(i,j)=|MVx(i,j)|+|MVy(i,j)|  (2)MA2=4·MD64+MD32  (3)

where MD64 is the number of blocks within one frame for MVD(i, j)≧64,and MD32 is the number of blocks within one frame for MVD(i, j)≧32.

(Moving Picture Conversion)

Disclosed next with reference to FIG. 2 is the first preferredembodiment of a moving-picture bitstream conversion apparatus accordingto the present invention. The conversion apparatus is suitable toconvert a bitstream transferred from the motion-compensated codingapparatus shown in FIG. 1 into another bitstream at a another transferrate.

A moving-picture bitstream transferred through an input terminal 21, forinstance, from the motion-compensated coding apparatus shown in FIG. 1,is supplied to a demultiplexer 22 and separated into a moving-picturebitstream and motion activity (MA). The moving-picture bitstream and themotion activity are supplied to a bitstream decimator 23 and adecimation controller 26, respectively.

B-picture bitstreams are removed from the moving-picture bitstreamsupplied to the bitstream decimator 23 based on a picture rate (ordecimation rate DR) set at the decimation controller 26. The remainingbitstream is supplied to a buffer 24 intermittently for each period inwhich B-pictures have been removed. The buffer 24 holds the intermittentinput and outputs a sequential bitstream at a low transfer bit rate inaccordance with the removed B-picture amount.

A picture rate (decimation rate) is set at the decimation controller 26for each segment between a P (I)-picture and the next P (I)-pictureduring which motion activity is detected.

The following are four types of picture rate (decimation rate) and thecorresponding decimation patterns for decimating only B-pictures forreproducing uniform pictures after decimation:

(1) 60 fps (0/5): no B-pictures decimated, or all pictures remain foreach segment;

(2) 30 fps (3/5): three B-pictures decimated, or two B-pictures remainfor each segment;

(3) 20 fps (4/5): four B-pictures decimated, or one B-picture remain foreach segment; and

(4) 10 fps (5/5): all (five) B-pictures decimated, or no B-picturesremain for each segment.

Five B-pictures in total are subjected to decimation in this embodimentso as to correspond to the motion-compensated coding apparatus shown inFIG. 1 in that 5-frame B-pictures are stored in the picture memory 8 fora 6-frame period until coding of one I- or P-picture per sixth frame isfinished, as disclosed.

FIG. 3 illustrates which B-pictures are decimated according to thepicture rate (decimation rate). B-picture(s) to be decimated is (are)decided so that the remaining B-picture(s) and the preceding and thefollowing I(P)-pictures after decimation are arranged at a constantinterval.

A moving-picture video signal mostly carries 60 frames (or fields)according to a limit on detection of a flicker on screen, however, suchnumber of frames (or fields) are not always required for every movingpicture. For example, cinema films require 24 frames per second (fps),so that un-smooth movement (jaggy movement, etc.) is noticeable only forrapidly moving scenes at 30 fps, however, not noticeable at 20 fpswhereas noticeable at 10 fps except stationary scenes.

Therefore, decimation control in which a picture rate (decimation rate)is switched among 60 fps only for rapid movement, 30 fps for frequentmovement, 20 fps for infrequent movement, and 10 fps for almoststationary scenes offers smooth and natural movement on screen whenmoving pictures are reproduced.

Discussed next is how a transfer rate varies according to B-picturedecimation as disclosed above.

Although depending on moving pictures, a code amount ratio of I-picturesto P-pictures is about 3:1 and that of P-pictures to B-pictures is alsoabout 3:1.

For example, a bitstream at bit rate of 6.0 Mbps before decimation iscomposed of I-pictures of 700 kbits, P-pictures of 200 kbits andB-pictures of 60 kbits on average.

When the 6.0-Mbps bitstream is decimated at a rate of 30 fps (3/5),three B-pictures per segment, or B-pictures of 180 kbits(60 kbits×3) aredecimated for 0.1 seconds, so that the bitstream becomes a 4.2-Mbpsbitstream.

When the 6.0-Mbps bitstream is decimated at a rate of 20 fps (4/5), fourB-pictures per segment, or B-pictures of 240 kbits (60 kbits×4) aredecimated for 0.1 seconds, so that the bitstream becomes a 3.6-Mbpsbitstream.

Moreover, when the 6.0-Mbps bitstream is decimated at a rate of 10 fps(5/5), five B-pictures per segment, or B-pictures of 300 kbits (60kbits×5) are decimated for 0.1 seconds, so that the bitstream becomes a3.0-Mbps bitstream.

It is apparent that a transfer rate is decreased to about 50% step bystep according to the picture rate (decimation rate) control. The amountof codes to be removed is, however, restricted in some extent forsegments of small motion activity at a low picture rate because suchsegments have a small amount of B-picture codes.

In the first embodiment, a picture rate (decimation rate) is controlledaccording to motion activity and also buffer occupancy (BO) on thebuffer 24 shown in FIG. 2. This is illustrated in FIG. 4. Bufferoccupancy closer to zero (the buffer 24 being almost empty) does notallow decrease in picture rate (decimation rate) even though motionactivity (MA) is relatively small. On the other hand, buffer occupancycloser to 1.0 (the buffer 24 being almost full) does not allow increasein picture rate (decimation rate) even though motion activity isrelatively large. A feed-back control is performed as shown in FIG. 2from the buffer 24 to the decimation controller 26 so that an outputcode rate becomes a predetermined value.

A recommendable transfer bit rate to be converted in this embodiment isabout 25% of an input transfer bit rate in accordance with acontrollable range and reproduced picture quality. For example, a6.0-Mbs input bitstream transfer rate is converted into about 4.5 Mbps.

An example of transfer rate conversion is illustrated in FIG. 5.Illustrated here is B-picture decimation controlled as shown in FIG. 4at buffer occupancy (BO) is 0.5 and 0.3. The nearer the BO to 1.0 (full)and the smaller the motion activity (MA), the more B-pictures aredecimated.

The bit stream conversion is performed from a fixed transfer bit rate toanother fixed transfer bit rate, as disclosed. Not only that, thebitstream conversion is performed, for example, from a variable transferbit rate to another variable transfer bit rate, from a fixed transferbit rate to a variable transfer bit rate and from a variable transferbit rate to a fixed transfer bit rate on asynchronous transfer mode(ATM) line. Such bitstream conversion on ATM line can be performedaccording to the embodiment shown in FIG. 2, which is apparent from theforegoing disclosure for a skilled in the art without a detaildescription.

Disclosed next with reference to FIG. 6 is the second preferredembodiment of a moving picture bitstream conversion apparatus accordingto the present invention.

Elements in this embodiment that are the same as or analogous toelements in the first embodiment (FIG. 2) are referenced by the samereference numbers.

This conversion apparatus is also suitable to convert a bitstreamtransferred from the motion-compensated coding apparatus shown in FIG. 1into another stream at a different transfer rate.

Moreover, this conversion apparatus is also suitable to conversion of abitstream transferred from a well-known motion-compensated codingapparatus with no motion activity detection because this conversionapparatus performs motion activity detection.

A moving-picture bitstream transferred through an input terminal 21, forinstance, from the motion-compensated coding apparatus shown in FIG. 1,is supplied to a bitstream decimator 23 and a demultiplexer 31.

A motion vector (MV) is separated from the bitstream supplied to thedemultiplexer 31 and supplied to a motion amount detector 32.

B-picture bitstreams are removed from the moving-picture bitstreamsupplied to the bitstream decimator 23 based on a picture rate (ordecimation rate DR) set at a decimation controller 26. The remainingbitstream is supplied to a buffer 24 intermittently for each period inwhich B-pictures have been removed. The buffer 24 holds the intermittentinput and outputs a sequential bitstream at a low transfer bit rate inaccordance with the removed B-picture amount.

The motion amount detector 32 detects motion activity (MA) based on themotion vector (MV) like the motion amount detector 15 shown in FIG. 1.However, an incoming bitstream caries intra-coded blocks with no motionvectors for I- and P-pictures. For I-pictures, the motion amountdetector 32 applies distance compensation to a motion vector of thepreceding B-picture to obtain motion activity. On the other hand, forintra-coded blocks of P-pictures, a predetermined relatively largemotion vector is applied thereto to obtain motion activity because suchblocks often carry moving pictures of rapid movement.

Like the first embodiment shown in FIG. 2, the decimation controller 26set a picture rate (or decimation rate) in accordance with the obtainedmotion activity. A decimation rate (frame rate) can be set in accordancewith motion activity (MA) and also buffer occupancy (BO) on the buffer24.

When a moving-picture bitstream is transferred from themotion-compensated coding apparatus shown in FIG. 1, the demultiplexer31 can separate motion activity from the input bitstream and supply itdirectly to the decimation controller 26.

(Bitstream Multiplexing)

A concept of the present invention on moving-picture bitstreammultiplexing is that a code amount control is performed aftermultiplexing a several moving-picture bitstreams transferred fromencoders, that include bitstreams of first pictures (I- or P-pictures inthe MPEG standard) to be used as reference pictures for inter-pictureprediction and bitstreams of second pictures other than the firstpictures, with decimation of the bitstreams of the second pictures(B-pictures in the MPEG standard) before multiplexing.

The bitstreams of the second pictures can be decimated with no affectionto reproduction of the bitstreams of the first pictures because thesecond pictures are not used as reference pictures for inter-pictureprediction. The present invention is therefore applicable to bitstreamsthat have already been coded, with no feedback of control information tomoving-picture encoders.

Another concept of the present invention on moving-picture bitstreammultiplexing is decimation control according to motion activity so thatdecimation is performed mainly on picture portions of a low motionactivity for which un-smooth motion will not be noticeable even though apicture rate is lowered by decimation rather than picture portions of ahigh motion activity for which un-smooth motion will be noticeable if apicture rate is lowered by decimation, for keeping picture quality whichwould otherwise be degraded.

Disclosed next with reference to FIG. 7 is the first preferredembodiment of a moving-picture bitstream multiplexing apparatusaccording to the present invention.

Bit streams AA and BA that are supplied to the multiplexing apparatusshown in FIG. 7 have been obtained by moving picture coding apparatus430 and 440, respectively. In detail, the moving picture codingapparatus 430 and 440 have processed moving pictures A and B,respectively, by encoding the pictures under the MPEG-2 standard at a6-frame segment between P- or I-pictures to obtain the bitstreams AA andBA, respectively, of 480 p (progressive) pictures (720×480 pixels, 60frames/sec.).

The bit streams AA and BA are supplied to bitstream decimators 200 and900 through input terminals 100 and 800, respectively. Each decimatordecimates B-pictures from the input bitstream at a picture rate (or adecimation rate) FR supplied by a decimation controller 600. The picturerate FR is set in the range from 60 fps to 10 fps (frame per second).

B-pictures are decimated according to the picture rate FR, the same asshown in FIG. 3. B-pictures to be decimated are decided so that theremaining B-pictures and the preceding and the following I(P)-picturesafter decimation are arranged at a constant interval.

The bitstreams AA and BA subjected to decimation as above are convertedinto bitstreams AB and BB and supplied to buffers 300 and 110,respectively. The buffers 300 and 110 hold intermittent (due todecimation) input of the bitstreams AB and BB, respectively, for severalframes, and supply the bitstreams to a multiplexer 400.

The multiplexer 400 receives bitstreams AB and BB at a ratio inproportion to their bit rates and multiplexes them to one bitstream. Thebitstreams AB and BB are multiplexed as bitstreams of almost the sametiming at a fixed total transfer rate of the bitstreams. The multiplexedbitstream at a fixed transfer rate is output through an output terminal500 and also supplied to a code amount observer 700.

The code amount observer 700 has a virtual buffer that corresponds to abuffer of an MPEG-decoder. The code amount observer 700 observes bufferoccupancy (BO) on the virtual buffer and varies the buffer occupancy bytaking out a bitstream from the virtual buffer for reproduction of eachframe, to output a value BO that has been normalized by the totalvirtual buffer capacity.

Discussed next is B-picture decimation control.

The decimation controller 600 controls a decimation rate (or a picturerate) for B-pictures, as listed below according to the normalizedvirtual buffer occupancy value BO. The picture rate is lowered as eachframe code amount becomes larger and the virtual buffer becomes emptywhereas it is raised as the code amount becomes smaller and the virtualbuffer becomes full.FR=60 fps . . . 0.6<BOFR=30 fps . . . 0.4<BO≦0.6FR=20 fps . . . 0.2<BO≦0.4FR=10 fps . . . BO≦0.2

The code amount is unchanged at a picture rate FR=60 fps, however, it isdecimated to almost 30%, 40% and 50% at FR=30 fps, 20 fps and 10 fps,respectively. A picture rate is almost 35 fps on average at a decimationrate of 25% on average.

Disclosed next with reference to FIG. 8 is the second preferredembodiment of a moving-picture bitstream multiplexing apparatusaccording to the present invention.

Elements in this embodiment that are the same as or analogous toelements in the first embodiment (FIG. 7) are referenced by the samereference numbers and will not be explained in detail.

Bitstreams AA and BA obtained as disclosed with respect to FIG. 7 aresupplied to code separators 210 and 240, respectively. Motion vectors(MVs) are detected for inter-picture prediction and supplied to motionamount detectors 220 and 250.

The motion amount detectors 220 and 250 detect motion activity (MA)based on the input motion vectors to be used for P-picture predictionfor each segment between a P-picture and the next P-picture for 0.1seconds. The detected motion activity are supplied to decimationcontrollers 230 and 260.

Motion activity is detected by using a horizontal component MVx (i, j)and a vertical component Mvy (i, j) of a motion vector used forP-picture coding in which “i” and “j” indicate horizontal and verticalpositions, respectively, on one frame when movement of one pixel is 1.0.Motion activity MA1 for the entire frame is obtained according to theexpression (1) already disclosed.

An incoming bitstream caries intra-coded blocks with no motion vectorsfor I- and P-pictures. For I-pictures, the motion amount detector 32applies distance compensation to a motion vector of the precedingB-picture to obtain motion activity. On the other hand, for intra-codedblocks of P-pictures, a predetermined relatively large motion vector isapplied thereto to obtain motion activity because such blocks oftencarry moving pictures of rapid movement.

The decimation controllers 230 and 260 performs picture rate FR(decimation rate) control and supply data FR(A) and FR(B) to bitstreamdecimators 200 and 900, respectively.

A moving-picture video signal mostly carries 60 frames (or fields)according to a limit on detection of a flicker on screen, however, suchnumber of frames (or fields) are not always required for every movingpicture. For example, cinema films require 24 frames per second (fps),so that un-smooth movement (jaggy movement, etc.). is noticeable onlyfor rapidly moving scenes at 30 fps, however, not noticeable at 20 fpswhereas noticeable at 10 fps except stationary scenes.

Therefore, picture rate (decimation rate) control in which the rate isswitched (even the virtual buffer occupancy BO is the same) among 60 fpsonly for rapid movement, 30 fps for frequent movement, 20 fps forinfrequent movement, and 10 fps for almost stationary scenes offerssmooth and natural movement on screen when moving pictures arereproduced.

The picture rate is decided as illustrated in FIG. 9 according to motionactivity (MA) and virtual buffer occupancy (BO). As illustrated, bufferoccupancy closer to zero (almost empty) allows decrease in picture rate(decimation rate) whereas BO closer to 1.0 (almost full) allows increasein picture rate (decimation rate). Lines “a”, “b” and “c” in FIG. 9 arethe borders between 60 fps and 30 fps, 30 fps and 20 fps, and 20 fps and10 fps, respectively.

Illustrated in FIG. 10 is B-picture decimation at BO =0.5. Eachbitstream is subjected to control according to motion activity (MA) toset frame (picture) rate FR (decimation rate) for B-picture decimation.Moving-picture bitstreams after multiplexing are controlled at apredetermined transfer rate.

As disclosed, according to the moving picture coding and moving-picturebitstream conversion in the present invention, motion activity of amoving picture is detected for controlling decimation of pictures whichwill not be used as reference pictures for inter-picture prediction,thus achieving a low code transfer rate (transfer bit rate) withmaintaining moving picture quality, such as, smooth movement on screen.

Moreover, according to the present invention, decimation of pictureswhich will not be used as reference pictures for inter-pictureprediction can be performed step by step, thus achieving fine picturerate adjustment for obtaining an optimum picture rate for human visualdetection capability on movement. The present invention therefore offersa bitstream of low transfer bit rate while maintaining high reproducedpicture quality.

Moreover, according to moving-picture bitstream multiplexing in thepresent invention, a code amount control is performed after multiplexinga several moving-picture bitstreams transferred from encoders, thatinclude bitstreams of first pictures (I- or P-pictures in the MPEGstandard) to be used as reference pictures for inter-picture predictionand bitstreams of second pictures other than the first pictures, withdecimation of the bitstreams of the second pictures (B-pictures in theMPEG standard) before multiplexing.

The bitstreams of the second pictures can be decimated with no affectionto reproduction of the bitstreams of the first pictures because thesecond pictures are not used as reference pictures for inter-pictureprediction. The present invention is therefore applicable to bitstreamsthat have already been coded, with no feedback of control information tomoving-picture encoders.

The present invention thus achieves an optimum code amount decrease, bymultiplexing, in bitstreams that have been coded at and transferred froma remote place and also bitstreams that have been coded and stored instorage media.

Furthermore, according to the present invention on moving-picturebitstream multiplexing, decimation is controlled according to motionactivity so that decimation is performed mainly on picture portions of alow motion activity for which un-smooth motion will not be noticeableeven though a picture rate is lowered by decimation rather than pictureportions of a high motion activity for which un-smooth motion will benoticeable if a picture rate is lowered by decimation. Thus, the presentinvention achieves a lower transfer bit rate for a multiplexed bitstreamwhile keeping picture quality which would otherwise be degraded.

1. A bitstream multiplexing apparatus for multiplexing a plurality ofincoming first moving-picture bitstreams into a moving-picture bitstreamat a bit rate lower than a total bit rate of the incoming firstmoving-picture bitstreams, each incoming first moving-picture bitstreamhaving first moving-picture bitstreams of first pictures used asreference pictures for inter-picture prediction and bitstreams of secondpictures other than the first pictures, the apparatus comprising: adecimation controller to set a decimation rate on the second pictures ofthe incoming first moving-picture bitstreams according to virtual bufferoccupancy to the multiplexed moving-picture bitstream; a bitstreamdecimator to decimate the bitstreams of the second pictures from theincoming first moving-picture bitstreams at a predetermined intervalaccording to the decimation rate, thus obtaining second bitstreams; amultiplexer to multiplex the second bitstreams to obtain the multiplexedbitstream; and virtual buffer means for obtaining a code amount of themultiplexed bitstream for each reproduction period as the virtual bufferoccupancy.
 2. The bitstream multiplexing apparatus according to claim 1,further comprising a motion amount detector to obtain motion activity oneach incoming first moving-picture bitstream, the motion activity beingused for setting the decimation rate.
 3. A method of multiplexing aplurality of incoming first moving-picture bitstreams into amoving-picture bitstream at a bit rate lower than a total bit rate ofthe incoming first moving-picture bitstreams, each incoming firstmoving-picture bitstream having first moving-picture bitstreams of firstpictures used as reference pictures for inter-picture prediction andbitstreams of second pictures other than the first pictures, the methodcomprising the steps of: obtaining a code amount of the multiplexedbitstream for each reproduction period as virtual buffer occupancy;setting a decimation rate on the second pictures of the incoming firstmoving-picture bitstreams according to the virtual buffer occupancy;decimating the bitstreams of the second pictures from the incoming firstmoving-picture bitstreams at a predetermined interval according to thedecimation rate, thus obtaining second bitstreams; and multiplexing thesecond bitstreams to obtain the multiplexed bitstream.
 4. Themultiplexing method according to claim 3, further comprising the step ofobtaining motion activity on each incoming first moving-picturebitstream, the motion activity being used for setting the decimationrate.